Network interconnect as a switch

ABSTRACT

An interconnect as a switch module (“ICAS” module) comprising n port groups, each port group comprising n−1 interfaces, and an interconnecting network implementing a full mesh topology where each port group comprising a plurality of interfaces each connects an interface of one of the other port groups, respectively. The ICAS module may be optically or electrically implemented. According to the embodiments, the ICAS module may be used to construct a stackable switching device and a multi-unit switching device, to replace a data center fabric switch, and to build a new, high-efficient, and cost-effective data center.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional application of U.S. application Ser.No. 16/921,264, filed on Jul. 6, 2020, which is the divisionalapplication of U.S. application Ser. No. 16/257,653 filed on Jan. 25,2019, which is a Continuation-in-Part application of U.S. applicationSer. No. 15/888,516 filed on Feb. 5, 2018, all of which are incorporatedherein by reference in their entireties, including any figures, tables,and drawings.

FIELD OF INVENTION

The present invention relates to computer network. In particular, thepresent invention relates to interconnecting structure of ICAS module,stackable switching device, multi-unit chassis switching device, networkpod, fanout cable transpose rack and datacenter network.

DISCUSSION OF THE RELATED ART

As a result of the recent rapid growth in application needs—in both sizeand complexity—today's network infrastructure is scaling and evolving ata high rate. The data traffic that flows from a data center to theInternet—i.e., “machine-to-user” traffic—is large, and ever increasing,as more people get connected, and as new products and services arecreated. However, machine-to-user traffic is merely “the tip of theiceberg,” when one considers the data traffic within the datacenter—i.e., “machine-to-machine” traffic—necessary to generate themachine-to-user data traffic. Generally, machine-to-machine data trafficis several orders of magnitude larger than machine-to-user data traffic.

The back-end service tiers and applications are distributed andlogically interconnected within a data center. To service each user whouses an application program (“app”) or a website, these back-end servicetiers and applications rely on extensive real-time “cooperation” witheach other to deliver the user's expected customized fast and seamlessexperience at the front end. To keep up with the demand, even though theinternal applications are constantly being optimized to improveefficiency, the corresponding machine-to-machine traffic grows at aneven faster rate than their continual optimization (e.g., at the currenttime, machine-to-machine data traffic is growing roughly faster thandoubling every year).

To be able to move fast and to support rapid growth are goals that areat the core of data center infrastructure design philosophy. Inaddition, the network infrastructure within a data center (“data centernetwork”) must be simple enough as to be managed by a small, highlyefficient team of engineers. It is desired that the data center networkevolves in the direction that makes deploying and operating the networkeasier and faster over time, despite scale and exponential growth.

Some of these applications needs relate to the increasing use of dataanalytic tools (“big data”) and artificial intelligence (“AI”), forexample. As discussed above, big data and AI have become verysignificant distributed applications. Servicing these applicationsrequire handling large amounts of data (e.g., petabytes), using greatcomputation power (e.g., petaflops), and achieving very low latency(e.g., responses that become available within 100 ns). Simultaneouslyproviding more powerful processors (“scaling-up”) and exploiting greaterparallel processing (“scaling-out”) have been the preferred approach toachieve performance. Unlike scientific computation, however, big-dataand AI applications are delivered in the form of services to largenumbers of users across the world. Thus, like web servers for webservices, clusters of servers dedicated to big data and AI applicationshave become significant parts of the data center network.

At the current time, data center networks have largely transitioned fromlayer-2 to all layer-3 (e.g., running Border Gateway Protocol (BGP) andEqual-cost Multi-Path (ECMP) protocols). A large-scale data center todayis typically operating at tens of petabits-per-second scale (petascale)and expects growth into the hundreds of petabits-per-second scale in thenear future. The cost of provisioning such data center ranges fromUS$300 million to US$1.5 billion.

Let us define several terms in Table 1, before proceeding with thedescription of this patent.

TABLE 1 Terminology Description media The media is a concept of aphysical entity. It can be optical or electronic. An interface is aconcept of a physical entity. It contains a transmission media and areception media. The media can be optical or electrical. An interfacecan associate with a MAC(Medium Access Control) entity or severalinterfaces can associate with a MAC entity. port A port is a concept ofa container entity. It includes a set of interfaces. The number of theinterfaces depends on the technology. For example, a 40G QSFP Ethernetport consists of 4 interfaces (a total of 8 fibers). The 40G QSFP portassociates with a MAC entity. The 10/40G optical technology isimplemented by reconfiguring the 40G QSFP Ethernet to 4 independent 10Ginterfaces each associates with a MAC entity. As such, each interfacecan provide connectivity and operate like a port. port group A portgroup is a concept of a container entity; it includes a set of ports;the number of ports depends on application. Each interface in a portgroup is configured to associate with a MAC presumably. In order to meetthe needs of description, this patent introduces and defines the term of“port group”. connection A connection consists of media and twointerfaces communicating through the media. link A link is a concept ofa container entity. It includes a set of connections. The number of theconnections depends on the technology. For example, two 40G QSFPconnectors docked together to form one link which contains 4connections; two 10/40G QSFP connectors docked together to form 4 linkseach contains 1 connection. downlink Refers to the link that connectstoward the hosts. uplink Refers to the link that connects toward thecore of the network. intralink Refers to the link that providesconnectivity inside a pod. interlink Refers to the link that providesconnectivity between two pods.

A review of our current state-of-the-art data center infrastructure isinstructive. In the following context, data or traffic aggregationrefers to multiplexing of communication frames or packets. Aggregationmodel and disaggregation model refer to topologies of communicationnetworks. The concept of data aggregation is orthogonal to the conceptof an aggregation or disaggregation model. Therefore, a disaggregationmodel can support data aggregation, as discussed below. A basic conceptin data communication is that communication channels can be error prone.Transmission over such communication channels at higher data rates andover large distances requires complex and costly transceivers.Consequently, channel encoding, error detection and correction, andcommunication protocols are many techniques to ensure data istransmitted over long distances with accuracy. In the past, as datatransmission was expensive, data aggregation (e.g., multiplexing fromdifferent data streams and multiplexing data from multiple topologicalsources) and data compression ensure even higher utilization of thecommunication channels and efficient management of the communicationcost. This is the origin of the aggregation (i.e., in both data andtopology) paradigm. This paradigm dominates the networking industry fordecades. Such aggregation is widely used in wide area networks (WANs),where transmission cost dominates over other network costs. Today'shardware architecture for data switching is also based on aggregation,i.e., each port is connected and aggregated from all other port. Intoday's communication networks, data is typically aggregated beforetransmitting on to “uplink” to connect to external network (e.g., theInternet), which tends to be the most expensive port of the dataswitching equipment. Due to both advances in semiconductor,fiber-optical, and interconnect technologies and economy of scale,network costs have reduced significantly. The aggregation model is notnecessarily the only—or the most suitable—solution in a data center. Intoday's data center networks, where machine-to-machine traffic(“east-west traffic”) dominates most of the bandwidth, being severalorders of magnitude than the machine-to-user bandwidth, multipathtopology and routing (ECMP) are deployed so that the combined networkbandwidth is large. However, traffic is still aggregated from allincoming port on to each outgoing port. Nonetheless, the multipathtopology signifies a disaggregation model. The detailed descriptionbelow places a structure and quantification onto the multipath topologyand discloses a disaggregation model, referred to herein as“interconnect as a Switch” (“ICAS”), which is significantly differentfrom the more traditional aggregation model for data centers.

Typically, in an enterprise or intranet environment, communicationpatterns are relatively predictable with a modest number of data sourcesand data destinations. These data sources and data destinations aretypically connected by a relatively small number of designated paths(“primary paths”), with some number of back-up or “secondary paths,”which are provided primarily for fault tolerance. In such anenvironment, the routing protocols of the enterprise network areoptimized to select a shortest single path between eachsource-destination pair in the absence of a failure.

Distributed computing frameworks (e.g., MapReduce, Hadoop and Dryad) andweb services (e.g., web search, ecommerce, social networking, dataanalytics, artificial intelligence and scientific computing) bring a newparadigm of computing that requires both interconnections between adiverse range of hosts and significant aggregate bandwidths. Due to thescarcity of ports even in the high-end commercial switches, a commonhierarchical network topology that has evolved is a fat tree withhigher-speed ports and increasing aggregate bandwidths, as one moves upthe hierarchy (i.e., towards the roots). The data center network, whichrequires substantial intra-cluster bandwidths, represents a departurefrom the earlier hierarchical network topology. In the multi-rootedtree, the shortest single-path routing protocol can significantlyunderutilize the available bandwidths. The ECMP is an improvement thatstatically stripes flows across available paths using flow hashingtechniques. ECMP is standardized in the IEEE 802.1Q Standard. ECMPallows “next-hop packet forwarding” to a single destination to occurover multiple “best paths,” as symmetric insuring flows on deterministicpaths. Equal cost multi-path routing can be used in conjunction withmost routing protocols, because it is a per-hop decision limited to asingle router. It can substantially increase bandwidth by load-balancingtraffic over multiple paths. When a data packet of a data stream arrivesat the switch, and multiple candidate paths are available for forwardingthe data packet to its destination, selected fields of the data packet'sheaders are hashed to select one of the paths. In this manner, the flowsare spread across multiple paths, with the data packets of each flowtaking the same path, so that the arrival order of the data packets atthe destination is maintained.

Note that ECMP performance intrinsically depends on both flow size andthe number of flows arriving at a host. A hash-based forwarding schemeperforms well in uniform traffic, with the hosts in the networkcommunicating all-to-all with each other simultaneously, or in whichindividual flow last only a few round-trip delay times (“RTTs”).Non-uniform communication patterns, especially those involving transfersof large blocks of data, do not perform well under ECMP without carefulscheduling of flows to avoid network bottlenecks.

In the detailed description below, the terms “fabric switch” and “spineswitch” are used interchangeably. When both terms appear in a network, afabric switch refers to a device in a network layer which is used formultipath networking among TOR devices, while a spine switch refers to adevice in a higher network layer which is used for multipath networkingamong pods.

A fat tree network suffers from three types of drawbacks—i.e., 1)congestion due to hash collision, 2) congestion due to an aggregationmodel, and 3) congestion due to a blocking condition. These congestionsare further examined in the following.

First, under ECMP, two or more large, long-lived flows can hash to thesame path (“hash collision”), resulting in congestion, as illustrated inFIG. 1a . FIG. 1a shows four fabric switches 10-0 to 10-3interconnecting five TOR switches 11-0 to 11-4. As shown in FIG. 1a ,each TOR switch has four ports each communicating with a port of one ofthe fabric switches 10-0 to 10-3, respectively. Each fabric switch hasfive ports each communicating with a port of one of TOR switches 11-0 to11-4, respectively. In FIG. 1a , two flows designating TOR switch 11-0are sourced from TOR switches 11-1 and 11-2. However, by chance, eachflow is hashed to a path that goes through fabric switch 10-0, whichcauses congestion at designating port 101 of fabric switch 10-0. (Ofcourse, the congestion problem could have been avoided if one of theflows is hashed to a path that goes through fabric switch 10-1 forinstance). Furthermore, the static mapping of the flows to paths byhashing does not consider either current network utilization or thesizes of the flows, so that the resulting collision overwhelms switchbuffers, degrade overall switch utilization, and increases transmissionlatency.

Second, in a fat tree network, the total bandwidth of the aggregatedtraffic may exceed the bandwidth of all the downlinks of all the fabricswitches facing the same TOR switch, resulting in aggregationcongestion, as shown in FIG. 1b . Such aggregation congestion is acommon problem in the aggregation model of today's switching network,and requires detailed rate limiting to avoid congestion. In FIG. 1b ,the traffic through the fabric switches 12-0 to 12-3 facing the TORswitch 13-0 is sourced from the TOR switches 13-1 to 13-4, but theaggregate traffic from the source (one source each from TOR switches13-1 to 13-3 and two sources from TOR switch 13-4) exceeds the combinedbandwidth of all the downlinks of all the fabric switches 12-0 to 12-3facing the TOR switch 13-0. More specifically, traffic is spread outevenly over fabric switch 12-1 to 12-3 without congestion; additionaltraffic from TOR switch 13-4 exceeds the downlink bandwidth of port 121of fabric switch 12-0 and thus causes congestion.

Third, there is a blocking condition called the “strict-sense blockingcondition,” which is applicable to statistically multiplexed flow-basednetworks (e.g., a TCP/IP network). The blocking condition results frominsufficient path diversity (or an inability to explore path diversityin the network) when the number and the size of the flows becomesufficiently large. FIG. 1c illustrates the blocking condition in a fattree network. As shown in FIG. 1c , the blocking condition occurs, forexample, when paths from fabric switches 14-0 and 14-1 to TOR switch15-0 are busy and paths from fabric switches 14-2 and 14-3 to TOR switch15-3 are busy, and a flow which requires a path through TOR switch 15-0arrives at TOR switch 15-3. An extra flow between TOR switch 15-0 and15-3 can take one of 4 possible paths. Say it takes the path from TORswitch 15-3 to fabric switch 14-1 and then from fabric switch 14-1 toTOR switch 15-0. However, the path from fabric switch 14-1 to TOR switch15-0 is busy already. Overall, multiplexing the blocked flow on to theexisting flows results in increased congestion, latency and/or packetloss.

At the same time as the demand on the data center network grows, therate of growth in CMOS circuit density (“Moore's law”) and the I/Ocircuit data rate appear to have slowed. The cost of lithographic andheat density will ultimately limit how many transistors can be packedinto a single silicon package. That is to say, an ultra large storage orcomputing system is bound to be achieved through multiple chips. It isunlikely that an ultra large system will be integrated on a single chipwith ultra-high integration density as in the past. The question thatarises here is how to build an ultra large bandwidth interconnectionbetween the chips. It is instructive to learn that a switching chipsoldered on printed circuit board (PCB) employs high-speed serialdifferential I/O circuit to transmit and receive data to/fromtransceiver module. A transceiver module interconnects to a transceivermodule on a different system to accomplish network communications. Anoptical transceiver performs the electrical-to-optical andoptical-to-electrical conversion. An electrical transceiver performscomplex electrical modulation and demodulation conversion. The primaryobstacle that hinders high-speed operation on PCB is thefrequency-dependent losses of the copper-based interconnection due toskin effects, dielectric losses, channel reflections, and crosstalk.Copper-based interconnection faces the challenge of bandwidth limit asthe data rate exceeds several tens of gigabit per second (Gb/s). Tosatisfy demands for bigger data bandwidth high-radix switch siliconintegrates hundreds of differential I/O circuits. For example, BroadcomTrident-II chip and Barefoot Network Tofino chip integrate 2×128 and2×260 differential I/O circuits for 10 Gb/s transmit and receiverespectively. To optimize system level port density, heat dissipationand bandwidth the I/O circuits and interfaces are gathered in groups andstandardized in specifications on electrical and optical properties. ForSFP+, each port has a pair of TX and RX serial differential interfacesat 10 Gb/s data rate. For QSFP, each port has four pairs of TX and RXserial differential interfaces at 10 Gb/s data rate each for a total of40 Gb/s or 4×10 Gb/s data rate. For QSFP28, each port has four pairs ofTX and RX serial differential interfaces at 25 Gb/s data rate each for atotal of 100 Gb/s or 4×25 Gb/s data rate. For QSFP-DD, each port haseight pairs of TX and RX serial differential interfaces with a data rateof 50 Gb/s data rate each for a total of 400 Gb/s or 8×50 Gb/s datarate. State of the art data centers and switch silicon employ 4 or 8interfaces (TX, RX) at 10 Gb/s or 25 Gb/s or 50 Gb/s per port as designconsiderations. These groupings are not necessarily unique. MTP/MPO asan optical interconnect standard defines up to 48 interfaces per portwhere each interface contains a pair of optical fibers one for transmitand one for receive. However the electrical and optical specificationsof transceiver with up to 48 interfaces per module are yet to come. Thedefinition of “port group” in this patent disclosure is extended toinclude more interfaces crossing multiple ports (e.g., 8 interfaces from2 QSFP's; 32 interfaces from 8 QSFP's, etc.). A person experienced inthe art can understand that this invention is applicable to otherinterconnect standards where multiple various number of interfaces otherthan 4 be grouped together in the future.

These limitations affect data center networks by, for example,increasing power consumption, slowing of performance increase, andincreasing procurement cycle. These developments exacerbate the powerneeds for the equipment, as well as their cooling, facility space, thecost of hardware, network performance (e.g., bandwidth, congestion, andlatency, management), and the required short time-to-build.

The impacts to network communication are several:

-   -   (a) The network industry may not have enough economy of scale to        justify CMOS technology of a smaller footprint;    -   (b) Simpler solutions should be sought to advance network        technology, rather than to create more complex ones and packing        more transistors;    -   (c) Scale-out solutions (i.e., in complement to scale-up        solution) should be sought to solve application problems (e.g.,        big-data, AI, HPC, and data center);    -   (d) The chip port density (i.e., the number of ports in the        traditional sense) can become flat¹; and    -   (e) Implementation of interfaces with signal rates in excess of        100 G will become increasingly difficult². ¹ Integration of        optical technology to the CMOS device may provide new        opportunity. However, do not expect a very high-radix chip,        which would allow network scalability, to emerge any time soon.²        One must think beyond the aggregation model (e.g., the        disaggregation model) to meet new network challenges.

Historically, high-speed networks have two classes of design space. Inthe first class of design space, HPC and supercomputing networkstypically adopt direct network topologies. In a direct network topology,every switch is connected to servers, as well as other switches in thetopology. Popular topologies include mesh, torus, and hypercube. Thistype of network is highly resource efficient and offers high capacitythrough numerous paths of various lengths between a source anddestination. However, the choice of which path to forward traffic overis ultimately controlled by proprietary protocols (i.e., non-minimumrouting) in switches, NICs, and by the end-host application logic. Thatis, an algorithm or manual configuration is required to achieve routing.Such routing protocols increase the burden on the developer and create atight coupling between applications and the network.

In the second class of design space, data centers scaling-out haveresulted in the development of indirect network topologies, such asfolded-Clos and multi-rooted trees (“fat trees”), in which servers arerestricted to the edges of the network fabric. The interior of thenetwork fabric consists of dedicated switches that are not connected toany servers, but simply route traffic within the network fabric. Datacenter networks of this type thus have a much looser coupling betweenapplications and network topology, placing the burden of path selectionon the network switches themselves. That is to say, based on Internetrouting technology such as BGP (Border Gateway Protocol) routingprotocol. The BGP routing protocol has a complete set of loopprevention, shortest path and optimization mechanisms. However, thereare strict requirements and restrictions on the network topology. Datacenter technology based purely on Internet BGP routing cannoteffectively support multipath with non-shortest path topologies. As aresult data center networks have traditionally relied on fat treetopologies, simple routing and equal cost multipath selection mechanisms(e.g., ECMP). It is precisely because data center routing technology hasrestrictions on the network topology. The benefits to datacenter fromnon-shortest multipath path network topology other than the equal costmultipath topology have not been explored in the past years ofdevelopments of the datacenter technologies.

The BGP and ECMP protocols are not without flaws. ECMP relies on statichashing of flows across a fixed set of shortest equal cost paths to adestination. For hierarchical topologies (e.g., fat tree), ECMP routinghas been largely sufficient when there are no failures. However, evennow direct network topologies (e.g., Dragonfly, HyperX, Slim Fly, BCube,and Flattened Butterfly), which employ paths of different lengths, havenot seen adoption in data centers because of the limitations imposed byboth commodity data center switches and the widespread adoption of ECMProuting in data center networks. ECMP is wasteful of network capacitywhen there is localized congestion or hot-spots, as it ignoresuncongested longer paths. Further, even in hierarchical networks, ECMPmakes it hard to route efficiently in the presence of failures, and whenthe network is no longer completely symmetric, and non-shortest pathsare available for improving network utilization.

FIG. 2a shows an architecture of a typical state-of-the-art data centernetwork, organized by three layers of switching devices—i.e.,“top-of-rack” (TOR) switches and fabric switches implemented in 96server pods 21-0 to 21-95 and spine switches implemented in 4 spineplanes 20-0 to 20-3—interconnected by interlinks in a fat tree topology.Details of a spine plane is shown in FIG. 2b where a spine planeconsists of 48 spine switches 22-0 to 22-47 each connecting to 96 serverpods. The connections from all 48 spine switches are grouped into 96interlinks each including a connection from one of spine switches 22-0to 22-47, respectively, for a total of 48 connections per interlink.Details of a server pod is shown in FIG. 2c , in which a server pod isshown to consist of 48 TOR switches 24-0 to 24-47 and 4 fabric switches23-0 to 23-3, with each TOR switch connected to all 4 fabric switches.Combining the connection information from FIG. 2b , server pod of FIG.2c may comprise 4 fabric switches 23-0 to 23-3 each connects one of 4spine planes by interlinks, respectively; each interlink comprising 48connections each connects one of 48 spine switches in a spine plane,respectively. Each TOR switch provides 48×10 G connections in 12×QSFPinterfaces as downlink to connect to servers. An edge pod is shown inFIG. 2d , details will be given in below.

As shown in FIGS. 2b and 2c , and in conjunction with FIG. 2a , the TOR,fabric and spine layers of switches include: (a) a TOR switch layerconsisting of 96×48 TOR switches which connect the servers in the datacenter and which are equally distributed over 96 “server pods”; (b) aspine switch layer consisting of 4×48 spine switches equally distributedover the 4 “spine planes”; and (c) a fabric layer consisting of 96×4fabric switches, also equally distributed over the 96 server pods. Inaddition, two of the server pods can be converted to two edge pods. FIG.2d shows an edge pod. As shown in FIG. 2d , edge pod 250 may comprise 4edge switches 25-0 to 25-3, each connects one of 4 spine planes byinterlinks, respectively; each interlink comprising 48 connections eachconnects one of 48 spine switches in a spine plane, respectively. Eachedge switch may include one or more uplinks that interconnect anexternal network.

Details of an implementation of the server and spine pods are furtherdescribed below in FIG. 2c, 2b in relation to FIG. 2a . Thisconfiguration facilitates modularity by assembling each fabric switchand spine switch in an 8 U multi-unit chassis with 96 QSFP ports. Asshown in FIG. 2c , each TOR switch is implemented by a switch with 16QSFP ports, which allocates 12 QSFP ports to connect to the servers in10 G interfaces (i.e., downlinks) and 4 QSFP ports to connect to thefour fabric switches in four 40 G interfaces in the same server pod. (Inthis detailed description, a QSFP represents a 40 Gbits/secondbandwidth, which can be provided in a single 40 G interface or four 10 Ginterfaces, each 40 G interface including four receive-transmit pairs ofoptical fibers and each 10 G interface including a receive-transmit pairof optical fibers). The 40 G interface between a TOR switch and a fabricswitch is used for both intra-pod and inter-pod data traffic.

Each fabric switch in a server pod is implemented by a 96 QSFP portsswitch, which allocates (i) 48 QSFP ports in 48 40 G interfaces with the48 TOR switches in the server pod in a fat tree topology, and (ii) 48QSFP ports in 48 40 G interfaces to the 48 spine switches in the singlespine plane the fabric switch is connected.

Each spine switch in a spine plane is also implemented by a 96 QSFPports switch, which provides all 96 QSFP ports in 96 40 G interfaceswith the 96 fabric switches connected to the spine plane, one from eachof the 96 server pods. The data traffic through the spine planerepresents inter-pod communications mostly for the server pods.

In the configuration of FIG. 2a , each server pod includes (i) 384 QSFPtransceivers, half of which are provided to the spine planes and half ofwhich are provided to the network side of the fabric switches, (ii) 192QSFP transceivers provided to the network side of the TOR switches,(iii) 576 transceivers provided to the servers; (iv) 192 optical QSFPcables, (v) 36 application-specific integrated circuits (ASICs), whichimplements the fabric switches and (vi) 48 ASICs, which implements theTOR switches. The ASIC suitable for this application may be, forexample, the Trident-II Ethernet Switch (“Trident II ASIC”). Each spineplane includes 4608 QSFP transceivers, 4608 optical QSFP cables and 432Trident II ASICs.

The implementation of FIG. 2a provides in practice improved congestionperformance but does not eliminate congestion. The network organizationis based on an aggregation model, intended to improve cost andutilization of communication ports and transmission media under theaggregation model. While this aggregation model may still be valuablefor wide-area networks (e.g., the Internet), recent advances ofsemiconductor technology and economic of scale have called thisaggregation model into question, when applied to local area networks.

SUMMARY

According to one embodiment of the present invention, an interconnect asa switch module (“ICAS” module) comprises n port groups, each portgroups comprising n−1 interfaces, and an interconnecting networkimplementing a full mesh topology where each port group comprising aplurality of interfaces each connects an interface of one of the otherport groups, respectively.

According to one embodiment of the present invention, a stackableswitching device is provided, which includes one or more ICAS modules asdepicted above, a plurality of switching devices, and a stackablerackmount chassis, each ICAS module being connected to the plurality ofswitching devices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection, while the restinterfaces of the at least some port groups for interconnectingdifferent switching devices are configured as interfaces for uplink. TheICAS module and the switching devices are housed in the stackablerackmount chassis.

One embodiment of the present invention provides a multi-unit switchingdevice, which includes: one or more ICAS modules implemented on a PCB asa circuit, a plurality of switching devices, and a multi-unit rackmountchassis, each ICAS module being connected to the plurality of switchingdevices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection, while the restinterfaces of the at least some port groups for interconnectingdifferent switching devices are configured as interfaces for uplink. TheICAS module and the switching devices are packaged in the multi-unitrackmount chassis.

According to one embodiment of the present invention, a network pod isdisclosed, which includes: a plurality of first layer switching devices,each having a plurality of interfaces for downlink interfaces configuredto receive and transmit data signals from and to a plurality of servers,and each having a plurality of network side interfaces divided into aplurality of interlinks and a plurality of intralinks, and the interlinkinterfaces being configured to connect to higher layer switchingdevices, and the intralink interfaces of the first layer switchingdevices each being configured and grouped into one or more port groups;and one or more second layer devices of ICAS modules whose interfacesare divided into intralink interfaces and uplink interfaces, and theintralink interfaces of an ICAS module being grouped into port groups toconnect to the corresponding port groups of the first layer switchingdevices, and each port groups of additional ICAS module being connectedto the additional port group of each of the first layer switches, andthe uplink interfaces are configured to connect to the external network.The first layer switching devices and the second layer devices areinterconnected to implement a full mesh network of a predeterminednumber of nodes.

K spine planes each having p interlinks are used to connect p networkpods each having k TOR switches. In a spine plane, k spine switchesinterconnect to a fanout cable transpose rack.

According to one embodiment of the present invention, a fanout cabletranspose rack may include: k first port groups connecting tocorresponding port groups of k spine switches through first plurality offiber optic cables; p second port groups through connecting secondplurality of fiber optic cables to form p interlinks. A plurality offanout cables are used to cross-connect the k first port groups and thep second port groups so that connections from all k spine switches aregrouped into p interlinks, each interlink including one connection fromeach spine switch, and each interlink having a total of k connections.

According to one embodiment of the present invention, a data centernetwork may have a plurality of interfaces for downlink configured toreceive and transmit data signals from and to a plurality of servers,and a plurality of interfaces for uplink configured to connect theInternet or connect another data center network with a similarconfiguration. The data center network may include: a group of networkpods (server pods/ICAS pods), each network pod in the group including:(a) a group of first layer switching devices, providing some interfacesas interfaces for downlink, and having the rest interfaces grouped intoone or more network side port groups; and (b) one or more second layerdevices, configured to interconnect at least some interfaces betweensome port groups of the first layer switching devices, wherein the restinterfaces of the some port groups for interconnecting the first layerswitching devices are configured as interfaces for uplink. The firstlayer switching devices and the second layer devices are interconnectedto implement a full mesh network of a predetermined number of nodes. Thenetwork pod further comprises a group of switch clusters, each includinga group of third layer switching devices, each of which routes aplurality of data signals received from or transmitted to acorresponding first layer switching device in each group of networkpods.

By simplifying the data center network infrastructure and reducinghardware requirement, the present invention addresses the problemsrelating to the power needs for the equipment and their cooling,facility space, the cost of hardware, network performance (e.g.,bandwidth, congestion, and latency, management), and the required shortbuilt time.

The present invention is better understood upon consideration of thedetailed description below in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a illustrates congestion due to hash collision in a fat treenetwork under ECMP.

FIG. 1b illustrates aggregation congestion in a fat tree networktopology.

FIG. 1c illustrates congestion due to blocking condition in a fat treenetwork.

FIG. 2a shows the architecture of a state-of-the-art data centernetwork.

FIG. 2b shows in detail an implementation of a spine plane of the datacenter network of FIG. 2 a.

FIG. 2c shows in detail an implementation of a server pod of FIG. 2ausing four fabric switches to distribute machine-to-machine trafficacross 48 top-of-rack switches.

FIG. 2d shows in detail an implementation of an edge pod of FIG. 2ausing four edge switches to provide interfaces for uplink to connect toexternal network.

FIG. 3 illustrates a “full mesh” topology in a network of 9 nodes.

FIG. 4a shows ICAS module 400, which interconnects 9 nodes, according tothe full mesh topology of FIG. 3.

FIG. 4b illustrates the connectivity between the internal interfaces andthe external interfaces of port group 7 of the 9-node ICAS module 400,in accordance with one embodiment of the present invention.

FIG. 5a shows ICAS module 500 connecting port group 2 of each of TORswitches 51-0 to 51-8 in a full mesh topology, in accordance with oneembodiment of the present invention.

FIG. 5b illustrates, in the full mesh topology network 500 of FIG. 5a ,port group 2 of TOR switch 51-1 routing a data packet to port group 2 ofTOR switch 51-7 through internal interface 52-1-7 of port group 50-1 andinternal interface 52-7-1 of port group 50-7 of ICAS2 module 500, inaccordance with one embodiment of the present invention.

FIG. 6a shows network 600, which is a more compact representation of thenetwork of FIG. 5 a.

FIG. 6b shows network 620, after additional ICAS modules are added tonetwork 600 of FIG. 6a , so as to provide greater bandwidth and pathdiversity.

FIG. 7a shows that, in the architecture of the data center of FIG. 2a ,the topology of a server pod may be reduced to a (4, 48) bipartitegraph.

FIG. 7b shows, as an example, network 720 represented as a (5, 6)bipartite graph.

FIG. 7c shows the 6-node full mesh graph embedded in the (5, 6)bipartite graph of FIG. 7 b.

FIG. 8a shows an improved data center network 800, in accordance withone embodiment of the present invention; data center networks 800includes 20 spine planes, providing optional uplinks 801, and 188 serverpods, providing optional uplinks 802, uplinks 801 and 802 connecting toone or more external networks.

FIG. 8b shows in detail an implementation of modified spine plane 820,having 20 spine switches, providing optional uplink 821 for connectingto an external network.

FIG. 8c shows in detail an implementation of modified server pod 830 ina (20, 21) fabric/TOR topology, having 20 fabric switches fordistributing machine-to-machine traffic across 20 top-of-rack switches,in accordance with one embodiment of the present invention; the 21^(st)TOR switch is removed from the modified server pod 830 so that theconnections are provided as optional uplink 831 for connecting thefabric switches to an external network.

FIG. 9a shows ICAS-based data center network 900, achieved by replacingthe server pods of network 800 of FIG. 8a (e.g., server pod 830 of FIG.8c ) with ICAS pods 91-0 to 91-197, each ICAS pod being shown in greaterdetail in FIG. 9c , according to one embodiment of the presentinvention; in FIG. 9a , optional uplinks 901, shared by 20 spine planes,and optional uplinks 902, shared by 188 ICAS pods are provided forconnecting to an external network.

FIG. 9b shows in detail spine plane 920, which implements one of thespine planes in data center network 900 and which is achieved byintegrating a fanout cable transpose rack into spine plane 820 of FIG.8b , according to one embodiment of the present invention; the spineswitches in spine plane 920 provide optional uplink 921 for connectingto an external network.

FIG. 9c shows in detail an implementation of ICAS pod 930, which isachieved by replacing fabric switches 83-0 to 83-19 in server pod 830 ofFIG. 8c , according to one embodiment of the present invention; eachICAS pod provides 20×10 G uplinks 932 for connecting to an externalnetwork.

FIG. 9d illustrates a spine switch implemented with a single chiphigh-radix (i.e., a high port count) switching integrated circuit; sucha spine switch makes use of the highest port count switching integratedcircuit available at present time.

FIG. 9e shows a spine switch formed by stacking together 4 switch boxesimplemented with a Trident-II ASICs (96×10 G configuration each) and 1ICAS box 953. ICAS box 953 combines 4 ICAS modules 95-0 to 95-3 in one 1U chassis. Each ICAS module contains 3 copies of ICAS1X5 configuration.Together the ICAS box 953 provides non-blocking 1:1 subscription ratioto each of the 4 switches 96-0 to 96-3.

FIG. 9f shows a spine switch of an ICAS-based multi-unit switchingdevice where 4 ICAS-based fabric cards 97-0 to 97-3 get connected in afull mesh topology to switching ASIC's 98-0 to 98-3. Switching ASIC 98-0and 98-1 are housed in line card 973, and switching ASIC's 98-2 and 98-3are housed in line card 974.

To facilitate cross-referencing among the figures and to simplify thedetailed description, like elements are assigned like referencenumerals.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention simplifies the network architecture by eliminatingthe switches in the fabric layer based on a new fabric topology,referred herein as the “interconnect-as-a-switch” (ICAS) topology. TheICAS topology of the present invention is based on the “full mesh”topology. In a full mesh topology, each node is connected to all othernodes. The example of a 9-node full mesh topology is illustrated in FIG.3. The inherent connectivity of a full mesh network can be exploited toprovide fabric layer switching.

As discussed in further detail below, the ICAS topology enables a datacenter network that is far superior to a network of the fat treetopology used in prior art data center networks. Unlike other networktopologies, the ICAS topology imposes a structure on the network whichreduces congestion in a large extent. According to one embodiment, thepresent invention provides an ICAS module as a component forinterconnecting communicating devices. FIG. 4a shows ICAS module 400,which interconnects 9 nodes according to the full mesh topology of FIG.3.

FIG. 4a shows ICAS module 400 having port groups 40-0 to 40-8 and eachport group providing 8 external interfaces and 8 internal interfaces. InICAS module 400, each of the internal interfaces of a port groupconnects an internal interface of one of the other port groups,respectively. In fact, each port group is connected to every one of theother port groups through exactly one internal interface. In thiscontext, each “interface” includes a receive-transmit pair of opticalfibers capable of, for example, a 10 Gbits per second data rate. In FIG.4a , the port groups are indexed as 0-8. Indexes can also be arbitraryunequal values (For example, these 9 port groups can also be indexed as5, 100, 77, 9, 11, 8, 13, 50, and 64). The 8 internal interfaces forthese 9 port groups are indexed according to the indexes of theconnected port groups (For example, the internal interfaces for the 7-thport group are 0, 1, 2, 3, 4, 5, 6 and 8 in the first example; and are5, 100, 77, 9, 11, 8, 13 and 64 in the second example). Furthermore,internal interface j of port group i is connected to internal interfacei of port group j. The external interfaces for each port group of ICASmodule 400 are indexed sequentially as 0-7.

FIG. 4b illustrates in detail the connectivity between the internalinterfaces and the external interfaces of a port group 7 in ICAS module400, in accordance with the present invention. As shown in FIG. 4b , inone embodiment, the external interfaces are connected one-to-one to theinternal interfaces sequentially in the index order (For example, forport group 7, external interfaces 42-0 to 42-7 are sequentiallyconnected to internal interfaces 41-0 to 41-6 and 41-8). Therefore, forport group i, external interfaces 0-7 are connected to internalinterfaces 0, . . . , i−1, i+1, . . . , and 8 respectively. Therefore,it can be easily seen that any pair of port groups x and y are connectedthrough internal interface x of port group y and internal interface y ofport group x. This indexing scheme allows an external switching deviceto assign routes for data packets using the internal interface indicesof the source port group and destination port group. No congestioncondition (e.g., due to hash collision, aggregation model, orstrict-sense blocking) can occur between any pair of port groups.

The internal interconnection between the port groups of the ICAS modulecan be realized via an optical media to achieve a full mesh structure.The optical media may be an optical fiber and/or 3D MEMS. The 3D MEMSuses a controllable micro-mirror to create an optical path to achieve afull mesh structure. In both of these implementations MPO connectors areused. Alternatively, the ICAS module may also be electricallyimplemented using circuits. In this manner, the port groups of the ICASmodule are soldered or crimped onto a PCB using connectors that supporthigh-speed differential signals and impedance matching. Theinterconnection between the port groups is implemented using a copperdifferential pair on the PCB. Since signal losses significantly varybetween different grades of high-speed differential connectors andbetween copper differential pairs on different grades of PCBs, an activechip is usually added at the back end of the connector to restore andenhance the signal to increase the signal transmission distance on thePCB. Housing the ICAS module in a 1 U to multi-U rackmount chassis willform a 1 U to multi-U interconnection device. The ICAS-basedinterconnection devices are then interconnected with switching devicesto form a full mesh non-blocking network. This novel network will beexplained in detail hereunder in a plurality of embodiments. When theICAS module of the 1 U to multi-U interconnection device is opticallyimplemented (based on optical fiber and 3D MEMS), MPO-MPO cables areused to connect the ICAS-based interconnection devices and the switchingdevices. When the ICAS module of the 1 U to multi-U interconnectiondevice is electrically implemented as circuits (based on PCB+chip), DACdirect connection cables or AOC active optical cables are used toconnect the ICAS-based interconnection devices and the switchingdevices.

As switching in ICAS module 400 is achieved passively by itsconnectivity, no power is dissipated in performing the switchingfunction. Typical port group-to-port group delay through an ICAS passiveswitch is around 10 ns (e.g., 5 ns/meter, for an optical fiber), makingit very desirable for a data center application, or for big data, AI andHPC environments.

The indexing scheme of external-to-internal connectivity in ICAS module400 of FIG. 4a is summarized in Table 2 below:

TABLE 2 ICAS Port Index of External Interface Group 0 1 2 3 4 5 6 7 0 12 3 4 5 6 7 8 1 0 2 3 4 5 6 7 8 2 0 1 3 4 5 6 7 8 3 0 1 2 4 5 6 7 8 4 01 2 3 5 6 7 8 5 0 1 2 3 4 6 7 8 6 0 1 2 3 4 5 7 8 7 0 1 2 3 4 5 6 8 8 01 2 3 4 5 6 7

FIG. 5a shows network 500, in which ICAS module 510 and port group 2 ofeach of TOR switches 51-0 to 51-8 interconnects in a full mesh topology,in accordance with one embodiment of the present invention.

As illustrated in FIG. 5b , in an ICAS module 510 in the full meshtopology network 500 of FIG. 5a , port group 51-1 of TOR switch 1 routesa data packet to port group 51-7 of TOR switch 7 through externalinterface 53-1-6 and internal interface 52-1-7 of port group 50-1 ofICAS module 510, and internal interface 52-7-1 and external interface53-7-1 of port group 50-7 of ICAS module 510, in accordance with oneembodiment of the present invention. As shown in FIG. 5b , TOR switch51-1, which is connected to port group 50-1 of ICAS module 510, receivesa data packet with a destination reachable through internal port group52-1-7 of ICAS module 510. TOR switch 51-1 has a port group thatincludes 8 interfaces 54-1-0 to 54-1-7 (provided as two QSFP ports)mapping one-to-one to external interfaces 53-1-0 to 53-1-7 of port group50-1 of ICAS module 510, which in turn maps one-to-one to internalinterfaces 52-1-0, 52-1-2 to 52-1-8 in sequential order of port group50-1 of ICAS module 510. TOR switch 51-7 has a port group that includes8 interfaces 54-7-0 to 54-7-7 (provided as two QSFP ports) mappingone-to-one to external interfaces 53-7-0 to 53-7-7 of port group 50-7 ofICAS module 510, which in turn maps one-to-one to internal interfaces52-7-0 to 52-7-6 and 52-7-8 in sequential order of port group 50-7 ofICAS module 510. Each interface in a TOR switch port may be a 10 Ginterface, for example. As port groups 50-1 and 50-7 of ICAS module 510are connected through the port groups' corresponding internal interfaces52-1-7 and 52-7-1, TOR switch 51-1 sends the data packet through itsinterface 54-1-6 to external interface 53-1-6 of ICAS module 510. Sincethe connectivity in the ICAS module 510 adopts a full mesh topology, thedata packet is routed to external interface 53-7-1 of ICAS module 510.

In full mesh topology network 500, the interfaces of each TOR switch isregrouped into port groups, such that each port group contains 8interfaces. To illustrate this arrangement, port group 2 from each TORswitch connects to ICAS module 510. As each TOR switch has a dedicatedpath through ICAS module 510 to each of the other TOR switches, nocongestion can result from two or more flows from different sourceswitches being routed to the same port of destination switch (the“Single-Destination-Multiple-Source Traffic Aggregation” case). In thatcase, for example, when TOR switches 51-0 to 51-8 each have a 10-G dataflow that has TOR switch 51-0 as destination, all the flows would berouted on paths through respective interfaces. Table 3 summarizes theseparate designated paths:

TABLE 3 ICAS ICAS Source Destination Source Internal InternalDestination T1.p2.c0 ↔ ICAS2.p1.c0 ↔ ICAS2.p0.c1 ↔ T0.p2.c0 T2.p2.c0 ↔ICAS2.p2.c0 ↔ ICAS2.p0.c2 ↔ T0.p2.c1 T3.p2.c0 ↔ ICAS2.p3.c0 ↔ICAS2.p0.c3 ↔ T0.p2.c2 T4.p2.c0 ↔ ICAS2.p4.c0 ↔ ICAS2.p0.c4 ↔ T0.p2.c3T5.p2.c0 ↔ ICAS2.p5.c0 ↔ ICAS2.p0.c5 ↔ T0.p2.c4 T6.p2.c0 ↔ ICAS2.p6.c0 ↔ICAS2.p0.c6 ↔ T0.p2.c5 T7.p2.c0 ↔ ICAS2.p7.c0 ↔ ICAS2.p0.c7 ↔ T0.p2.c6T8.p2.c0 ↔ ICAS2.p8.c0 ↔ ICAS2.p0.c8 ↔ T0.p2.c7

In other words, in Table 3, the single-connection data between firstlayer switch i connected to the port group with index i and first layerswitch j connected to the port group with index j is directlytransmitted through the interface with index j of the port group withindex i and the interface with index i of the port group with index j.

In Table 3 (as well as in all Tables herein), the switch source and theswitch destination are each specified by 3 values: Ti.p_(j).c_(k), whereT_(i) is the TOR switch with index i, p_(j) is the port group with indexj and c_(k) is the interface with index k. Likewise, the sourceinterface and destination interface in ICAS module 500 are also eachspecified by 3 values: ICASj.p_(i).c_(k), where ICASj is the ICAS modulewith index j, p_(i) is the port group with index i and c_(k) is theinternal or external interface with index k.

An ICAS-based network is customarily allocated so that when its portgroups are connected to port group i from all TOR switches the ICAS willbe labeled as ICASi with index i.

Congestion can also be avoided in full mesh topology network 500 with asuitable routing method, even when a source switch receives a largeburst of aggregated data (e.g., 80 Gbits per second) from all itsconnected servers to be routed to the same destination switch (the“Port-to-Port Traffic Aggregation” case). In this case, it is helpful toimagine the TOR switches as consisting of two groups: the source switchi and the rest of the switches 0 to i−1, i+1 to 8. The rest of theswitches are herein collectively referred to as the “fabric group”.Suppose TOR switch 51-1 receives 80 Gbits per second (e.g., 8 10 Gflows) from all its connected servers all designating to destination TORswitch 51-0. The routing method for the Port-to-Port Traffic Aggregationcase allocates the aggregated traffic to its 8 10 G interfaces with portgroup 51-1 as in FIG. 5a , such that the data packets in each 10 Ginterface is routed to a separate TOR switch in the fabric group (Table4A):

TABLE 4A ICAS ICAS Source Destination Source Internal InternalDestination T1.p2.c0 ↔ ICAS2.p1.c0 ↔ ICAS2.p0.c1 ↔ T0.p2.c0 T1.p2.c1 ↔ICAS2.p1.c2 ↔ ICAS2.p2.c1 ↔ T2.p2.c1 T1.p2.c2 ↔ ICAS2.p1.c3 ↔ICAS2.p3.c1 ↔ T3.p2.c1 T1.p2.c3 ↔ ICAS2.p1.c4 ↔ ICAS2.p4.c1 ↔ T4.p2.c1T1.p2.c4 ↔ ICAS2.p1.c5 ↔ ICAS2.p5.c1 ↔ T5.p2.c1 T1.p2.c5 ↔ ICAS2.p1.c6 ↔ICAS2.p6.c1 ↔ T6.p2.c1 T1.p2.c6 ↔ ICAS2.p1.c7 ↔ ICAS2.p7.c1 ↔ T7.p2.c1T1.p2.c7 ↔ ICAS2.p1.c8 ↔ ICAS2.p8.c1 ↔ T8.p2.c1

Note that the data routed to TOR switch 51-0 has arrived at itsdesignation and therefore would not be routed further. Each TOR switchin the fabric group, other than TOR switch 51-0, then allocates itsinterface 0 for forwarding its received data to TOR switch 51-0 (Table4B3):

TABLE 4B ICAS ICAS Source Destination Source Internal InternalDestination — ↔ — ↔ — ↔ — T2.p2.c0 ↔ ICAS2.p2.c0 ↔ ICAS2.p0.c2 ↔T0.p2.c1 T3.p2.c0 ↔ ICAS2.p3.c0 ↔ ICAS2.p0.c3 ↔ T0.p2.c2 T4.p2.c0 ↔ICAS2.p4.c0 ↔ ICAS2.p0.c4 ↔ T0.p2.c3 T5.p2.c0 ↔ ICAS2.p5.c0 ↔ICAS2.p0.c5 ↔ T0.p2.c4 T6.p2.c0 ↔ ICAS2.p6.c0 ↔ ICAS2.p0.c6 ↔ T0.p2.c5T7.p2.c0 ↔ ICAS2.p7.c0 ↔ ICAS2.p0.c7 ↔ T0.p2.c6 T8.p2.c0 ↔ ICAS2.p8.c0 ↔ICAS2.p0.c8 ↔ T0.p2.c7

In other words, at least one multi-connection data between the firstlayer switch i connected to the port group indexed i and the first layerswitch j connected to the port group indexed j is transmitted throughthe first layer switches connected to at least one of the port groupsother than the port group with source index. The multi-connection dataarriving at the destination switch will cease to be further routed andtransmitted.

To put it more precisely, the multi-connection data transmissionoccurring between first layer switch i connected to the port group withindex i and first layer switch j connected to the port group with indexj includes the transmissions includes: as in Table 4A, the first layerswitch i is connected, via a plurality of interfaces of the port groupwith a plurality of index i, to a plurality of first layer switches witha plurality of corresponding indexes for transmission; as in Table 4B, aplurality of the first layer switches with the indexes as shown areconnected, via interfaces with index j of the port groups, to theinterfaces with the indexes as shown of the port groups with index j ofthe first layer switches for transmission; those transmissions thatarrive at a destination switch will stop routing.

Thus, the full mesh topology network of the present invention providesperformance that is in stark contrast to prior art network topologies(e.g., fat tree), in which congestions in the fabric switch cannot beavoided under Single-Destination-Multiple-Source Traffic Aggregation andPort-to-Port Traffic Aggregation cases.

Also, as discussed above, when TOR switches 51-0 to 51-8 abide by therule m≥2n−2, where m is the number of network-side interfaces (e.g., theinterfaces with a port group in ICAS module 500) and n is the number ofthe TOR switch's input interfaces (e.g., interfaces to the serverswithin the data center), a strict blocking condition is avoided. Inother words, a static path is available between any pair of inputinterfaces under any traffic condition. Avoiding such a blockingcondition is essential in a circuit-switched network, but is notnecessarily significant in a flow-based switched network.

In the full mesh topology network 500 of FIG. 5a , each port group with8 interfaces of ICAS module 500 connects to a port group with 8interfaces (e.g., 8 10-G interfaces) of a corresponding TOR switch. Fullmesh topology network 500 of FIG. 5a may be redrawn in a more compactform in FIG. 6a , with a slight modification. FIG. 6a illustrates ICAS2module 60-2 interconnecting to port group 2 of each of TOR switches 61-0to 61-8. In FIG. 6a , the interfaces between port group 2 of TOR switch61-0 and port group 0 of ICAS module 60-2 (now labeled ‘ICAS2’) arerepresented as a single line (e.g., the single line between port group 2of TOR switch 61-0 and port group 0 of ICAS module 60-2). Such a line,of course, represents all 8 eight interfaces between the TOR switch anda corresponding port group in ICAS module 60-2. This is exactly the casein FIG. 6b where each TOR switch 63-0 to 63-8 is shown also to have 4port groups, to allow configuring network 620 of FIG. 6b , where threeadditional ICAS modules 62-0, 62-1 and 62-3 in addition to 62-2 andcorresponding interfaces are added to network 600 of FIG. 6 a.

In full mesh topology network 500, uniform traffic may be spread out tothe fabric group and then forwarded to its destination. In network 620of FIG. 6b , the additional ICAS modules may be used to provide greaterbandwidth. So long as the additional port groups are available in theTOR switches, additional ICAS modules may be added to the network toincrease path diversity and bandwidth.

The inventor of the present invention investigated in detail thesimilarities and the differences between the full mesh topology of thepresent invention and other network topologies, such as the fat treetopology in the data center network of FIG. 2a . The inventor firstobserves that, in the architecture of the data center network of FIG. 2a, the fat tree network represented in a server pod (the “fabric/TORtopology”) can be reduced to a (4, 48) bipartite graph, so long as thefabric switches merely perform an interconnect function for trafficoriginated among the TOR switches. This (4, 48) bipartite graph is shownin FIG. 7a . In FIG. 7a , the upper set of nodes, nodes 0-3 (“fabricnodes”) 70-0 to 70-3, represent the four fabric switches in the serverpod of FIG. 2a and the lower set of 48 nodes (i.e., leaf 0-47), labeled71-0 to 71-47, represent the 48 TOR switches in a server pod of FIG. 2a.

The inventor discovered that an n-node full mesh graph is embedded in afabric-leaf network represented by a bipartite graph with (n−1, n) nodes(i.e., a network with n−1 fabric nodes and n TOR switch leaves). FIG. 7bshows, as an example, a (5, 6) bipartite graph with 5 nodes 72-0 to 72-4and 6 leaves 73-0 to 73-5. FIG. 7c shows the 6-node full mesh graph 740with 6 nodes 74-0 to 74-5 embedded in the (5, 6) bipartite graph of FIG.7 b.

This discovery leads to the following rather profound results:

(a) An n-node full mesh graph is embedded in an (n−1, n)-bipartitegraph; and the (n−1, n) bipartite graph and the data center Fabric/TORtopology have similar connectivity characteristics;

(b) A network in the (n−1, n) Fabric/TOR topology (i.e., with n−1 fabricswitches and n TOR switches) can operate in same connectivitycharacteristics as a network with full mesh topology (e.g., network 500of FIG. 5a );

(c) Fabric switches are unnecessary in an (n−1, n) Fabric/TOR topologynetwork, as the fabric switches merely performs interconnecting functionamong the TOR switches (i.e., these fabric switches can be replaced bydirect connectivity among TOR switches);

(d) A data center network based on a fat tree topology (e.g., theFabric/TOR topology) can be improved significantly using ICAS modules.

In the following, a data center network that incorporates ICAS modulesin place of fabric switches may be referred to as an “ICAS-based” datacenter network. An ICAS-based data center network has the followingadvantages:

(a) less costly, as fabric switches are not used;

(b) lower power consumption, as ICAS modules are passive;

(c) less congestion;

(d) lower latency;

(e) effectively less network layers (2 hops less for inter-pod traffic;1 hop less for intra-pod traffic);

(f) greater scalability as a data center network.

These results may be advantageously used to improve typicalstate-of-the-art data center networks. FIG. 8a shows an improved datacenter network 800, in accordance with one embodiment of the presentinvention. Data center network 800 uses the same types of components asthe data center network of FIG. 2a (i.e., spine switches, fabricswitches and TOR switches), except that the number of fabric switchesare increased to one less than the number of TOR switches (FIG. 8c showsequal number of fabric switches and TOR switches because one of the TORswitch, the 21^(st) TOR switch, is removed so that the 20 interfacesconnected to it from the 20 fabric switches are provided as uplink toconnect to external network).

FIG. 8a shows the architecture of an improved data center network,organized by three layers of switching devices—i.e., “top-of-rack” (TOR)switches and fabric switches implemented in 188 server pods 81-0 to81-187 and spine switches implemented in 20 spine planes 80-0 to80-19—interconnected by interlinks in a fat tree topology. An interlinkrefers to the network connections between a server pod and a spineplane. For example, interlink k of each of the 188 server pods isconnected to spine plane k; interlink p of each of the 20 spine planesis connected to server pod p. The 20 spine planes each provide anoptional uplink (e.g. uplink 801) and the 188 server pods each providean optional uplink (e.g., uplink 802) for connection to one or moreexternal networks. In this example, to allow comparison, the numbers ofserver pods and spine plane are chosen so that the improved data centernetwork 800 and the state-of-the-art data center network 200 have thesame network characteristics (2.2 Pbps total server-side bandwidth; 3:1oversubscription ratio—server-side to network-side bandwidth ratio;Trident-II ASIC). Other configurations of the improved data centernetwork are also possible, for instance, 32-TOR server pod or 48-TORserver pod but with higher radix switching silicon than the Trident-IIASIC.

Details of a spine plane of FIG. 8a are shown in FIG. 8b . In FIG. 8b ,spine plane 820 consists of 20 spine switches 82-0 to 82-19 eachconnecting to 188 server pods. The connections from all 20 spineswitches are grouped into 188 interlinks, with each interlink includinga connection from each spine switch 82-0 to 82-19, for a total of 20connections per interlink.

Details of a server pod of FIG. 8a are shown in FIG. 8c . In FIG. 8c ,the network-side connection (as opposed to the server-side connection)of the server pod is separated into intra-pod links and inter-pod links(i.e., the interlinks). The two types of links are made independent fromeach other. The intra-pod region 832 consists of the intra-pod links,the 20 TOR switches 84-0 to 84-19 and the 20 fabric switches 83-0 to83-19 interconnected by the intra-pod links in a fat tree topology. Forexample, connection k in each of the 20 TOR switches is connected tofabric switch k; connection p of each of the 20 fabric switches isconnected to TOR switch p. 20 fabric switches each provide an optionaluplink (e.g., uplink 831) to connect to an external network. Theinter-pod region consists of the inter-pod links (i.e., the interlinks)and 20 TOR switches 84-0 to 84-19 on the interlink side. Each interlinkprovides 20 10 G connections to connect to all 20 spine switches on thesame spine plane. Each server pod includes a total of 20 links. Forexample, interlink k of each of the 188 TOR switches across the 188server pods are connected to spine plane k; interlink p of each of the20 spine planes are connected to server pod p. Each TOR switch provides48×10 G connections in 12×QSFP ports as downlink to connect to servers.

The data traffic through the fabric switches is primarily limited tointra-pod. The TOR switches now route both the intra-pod traffic as wellas inter-pod traffic and are more complex. The independent link typesachieve massive scalability in data center network implementations.(Additional independent links provided from higher radix switching ASICmay be created to achieve larger scale of connectivity objectives).Additionally, data center network 800 incorporates the full meshtopology concept (without physically incorporating an ICAS module) toremove redundant network devices and allow the use of innovativeswitching methods, in order to achieve a “lean and mean” data centerfabric with improved data traffic characteristics.

As shown in FIG. 8c , FIG. 8b and FIG. 8a , data center network 800includes 20×188 TOR switches and 20×188 fabric switches equallydistributed over 188 server pods, and 20×20 spine switches equallydistributed over 20 spine planes. In FIG. 8a , each TOR switch has 10010 G-connections (i.e., 25 QSFPs of bandwidth in 10 G mode), of which 6010 G-connections are provided server-side and 40 10 G-connections areprovided network-side. (Among the network-side connections 20 10G-connections are used for intra-pod traffic and 20 10 G-connections areused for inter-pod traffic). In each server pod, fabric switches 83-0 to83-19 each include 21 10 G-connections, of which 20 10 G-connections areallocated to connect with a 10 G-connection in each of TOR switches 84-0to 84-19, and the rest being converted to provide as uplink to connectto external network. In this manner, fabric switches 83-0 to 83-19support the intra-pod region data traffic and the uplinks in the serverpod by a 21-node full mesh topology (with the uplinks of fabricsswitches 0-19 collectively seen as one node). Using a suitable routingalgorithm, such as any of those described above in conjunction withSingle-Source-Multiple-Destination Traffic Aggregation and Port-to-PortTraffic Aggregation, network congestion can be eliminated from allfabric switches.

As the network in the intra-pod region of each server pod can operate inthe same connectivity characteristics as a full mesh topology network,all the 20 fabric switches of the server pod may be replaced by an ICASmodule. ICAS-based data center network 900, resulting from substitutingfabric switches 83-0 to 83-19 of data center network 800, is shown inFIG. 9a . To distinguish from the server pod of data center network 800,a server pod with its fabric switches replaced by an ICAS module isreferred to as an “ICAS pod.”

FIG. 9a shows the architecture of an ICAS-based data center network,organized by three layers of devices—i.e., “top-of-rack” (TOR) switches,ICAS module implemented in 188 server pods 91-0 to 91-187 and spineswitches implemented in 20 spine planes for 90-0 to 90-19—interconnectedby interlinks in a fat tree topology. 20 spine planes provide optionaluplinks 901 and 188 ICAS pods provide optional 188×20×10 G uplinks 902for connecting to an external network. The number of network devices inthe data center network should be interpreted as illustrative only.

Details of a spine plane of FIG. 9a are shown in FIG. 9b according toone embodiment. In FIG. 9b , spine plane 920 includes 20 spine switches92-0 to 92-19 and a fanout cable transpose rack 921. The fanout cabletranspose rack contains: k first port groups 923 are connected tocorresponding port groups of k spine switches through a plurality offirst MPO-MPO fiber cables, where each first port group including ┌p/m┐first MPO adapters, and each first MPO adapter including m interfaces(where each interface includes one transmit fiber channel and onereceive fiber channel), and a plurality of first MPO fiber adapters fromthe k port groups 923 are connect to LC optical fiber adapter mountingpanel 922 through a plurality of first MPO-LC fanout fiber cables, wherek=20, p=188, m=4, and ┌ ┐ is a ceiling function; the fanout cabletranspose rack 921 includes p second port groups 924 that are connectedto a plurality of second MPO-MPO fiber cables to form interlinks 99-0 to99-187, each second port group contains ┌k/ml┐ second MPO fiberadapters, each of which includes m interfaces (where each interfaceincludes one transmit fiber channel and one receive fiber channel), anda plurality of second MPO fiber adapters from the p port groups 924 areconnected to LC optical fiber adapter mounting panel 922 through aplurality of the second MPO-LC fanout cables; a plurality of firstMPO-LC fanout fiber cables cross-connect a plurality of second MPO-LCfanout fiber cables on the LC fiber adapter mounting panel 922, throughcross-connection, all connections from k spine switches 92-0 to 92-19are reorganized into p interlinks 99-0 to 99-187, each interlinkincludes one connection from each of the spine switches 92-0 to 92-19,each interlink contains k connections in total.

That is, on one side of the fanout cable transpose rack 921 is k firstport groups 923, each first port group has ┌p/m┐ of first MPO adapters,where ┌ ┐ is a ceiling function, each port groups connects to acorresponding port group of a spine switch through the ┌p/m┐ firstMPO-MPO cables. On the other side of the fanout cable transpose rack 921is p second port groups 924, each second port group has ┌k/m┐ of secondMPO adapters, where ┌ ┐ is an ceiling function, each port group connectsto ┌k/m┐ second MPO-MPO cables to form an interlink to the ICAS pod.

As pointed out earlier in this detailed description, thestate-of-the-art data centers and switch silicon are designed with 4interfaces (TX, RX) at 10 Gb/s or 25 Gb/s each per port in mind.Switching devices are interconnected at the connection level inICAS-based data center. In such a configuration, a QSFP cable coming outfrom a QSFP transceiver is separated into 4 interfaces, and 4 interfacesfrom different QSFP transceivers are combined in a QSFP cable forconnecting to another QSFP transceiver. Also, a spine plane mayinterconnect a large and varying number of ICAS pods (e.g., in thehundreds) because of the scalability of an ICAS-based data centernetwork. Such a cabling scheme is more suitable to be organized in afanout cable transpose rack (e.g., fanout cable transpose rack 921),which may be one or multiple racks and be integrated into the spineplanes. Specifically, the spine switches and the TOR switches may eachconnect to the fanout cable transpose rack with QSFP straight cables.Such an arrangement simplifies the cabling in a data center. FIG. 9billustrates such an arrangement for data center network 900 of FIG. 9 a.

In the embodiment shown in FIG. 9b , the first and the second opticalfiber adapters are MPO adapters, the first and the second cables areMPO-MPO cables, the first and the second fanout cables are MPO-LC fanoutcables, the mounting panel is LC optical fiber adapter mounting panel.One skilled in the art would understand that different types of opticalfiber adapters/cable/optical fiber adapter mounting panel may also beused, such as FC, SC, LC, and MU.

Details of an ICAS pod of FIG. 9a are shown in FIG. 9c . In FIG. 9c ,the network side interface (as opposed to the server-side interface) ofan ICAS pod is divided into intra-pod links (i.e. intralinks) andinter-pod links (i.e., interlinks) and the two types of links are madeindependent from each other. The intra-pod region consists of intralinksbetween the 20 TOR switches 93-0 to 93-10 19 and ICAS module 931,interconnected by 10 G connections in a full mesh topology. Each ICASmodule may provide 20 10 G uplinks 932 to connect to one or moreexternal networks. The inter-pod region consists of interlinks. ICAS podmay comprise 20 TOR switches 93-0 to 93-19 each connects one of 20 spineplanes by interlinks, respectively; each interline comprising 20connections each connects one of 20 spine switches in a spine plane,respectively. For example, interlink k of each of 188 TOR switchesacross the 188 ICAS pods is connected to spine plane k; interlink p ofeach of the 20 spine planes is connected to server pod p. Each TORswitch provides 60×10 G interfaces in 15×QSFP ports as a downlink forconnecting to servers.

The data traffic through the ICAS module is primarily limited tointra-pod. The TOR switches now perform routing for the intra-podtraffic as well as inter-pod traffic and are more complex. Theindependent link types achieve massive scalability in data centernetwork implementations. (Additional independent link provided fromhigher radix switching ASIC may be created to achieve a larger scale ofconnectivity objectives).

As shown in FIG. 9c , FIG. 9b and FIG. 9a , each TOR switch allocates20×10 G-interfaces (5×QSFPs in 10 G mode) to connect to its associatedICAS module (e.g., ICAS module 931) to support intra-pod traffic, and 5QSFPs in 10 G mode (20 10 G-interfaces) to connect to the fibertranspose rack to support inter-pod traffic. As shown in FIG. 9c , eachICAS pod includes 20×5 QSFP transceivers for intra-pod traffic,connected by 100 QSFP straight cables, and 20×15 QSFP (10 G mode)transceivers for server traffic, for a total 500 QSFP transceivers. The20 TOR switches in an ICAS pod may be implemented by 20 Trident IIASICs. Although 20 TOR switches are shown in each ICAS pod in FIG. 9c ,the ICAS module is scalable to connect up to 48 TOR switches in an ICASpod (based on 32×QSFP Trident-II+ switch ASIC).

Together, the ICAS pods and the spine planes form a modular networktopology capable of accommodating hundreds of thousands of 10G-connected servers, scaling to multi-petabit bisection bandwidth, andcovering a data center with congestion improved and non-oversubscribedrack-to-rack performance.

According to one embodiment of the present invention, a spine switch canbe implemented using a high-radix (e.g., 240×10 G) single chip switchingdevice, as shown in FIG. 9d . Single-chip implementation saves the costof extra transceivers, cables, rack space, latency and power consumptionthan multi-unit (rack unit) chassis-based switching device and stackableswitching device implementations. The disadvantage of the single-chipspine switch approach is its network scalability, which limits thesystem to 240 ICAS pods at this time. As mentioned above, thesemiconductor implementation limits the scale of a high-radix switchingintegrated circuit.

To overcome the limitation on the port count of the silicon chip, one ormore 1 U to multi-U rackmount chassis each packaged with one or moreICAS modules, and a plurality of 1 U rackmount chassis each packagedwith one or more switching devices, can be stacked up in one or moreracks, interconnected, to form a higher-radix (i.e. high network portcount) stackable spine switching device (e.g., ICAS-based stackableswitching device). Each ICAS module is connected to the plurality ofswitching devices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection. The interfaces of therest of the at least some port groups for interconnecting differentswitching devices are configured as an uplink. When theICAS-module-based 1 U to multi-U rackmount chassis are opticallyimplemented (based on optical fiber and 3D MEMS), MPO-MPO cables may beused to connect the ICAS-based interconnection devices and the switchingdevices. When the ICAS-module-based 1 U to multi-U rackmount chassis areelectrically implemented as circuits (based on PCB+chip), DAC directconnection cables or AOC active optical cables may be used to connectthe ICAS-based interconnection devices and the switching devices.

Details of an ICAS-based stackable switching device 950 are shown inFIG. 9e . FIG. 9e shows ICAS modules 95-0 to 95-3 each connected in afull mesh topology to switches 96-0 to 96-3. In one embodiment, 4Trident-II ASIC-based switches 96-0 to 96-3, each having a switchingbandwidth of 24 QSFPs in 10 G mode provided in 1:1 subscription ratio,and an ICAS box 953 integrating 4 ICAS modules 95-0 to 95-3 in one 1 Uchassis and each ICAS module containing 3 duplicate copies of ICAS1X5sub-modules and each sub-module providing 4×10 G of uplink 951 may beused to builds a stackable spine switch, as shown in FIG. 9e . The 4switches 96-0 to 96-3 provide ports 952 of 1.92 Tbps of bandwidth toconnect to servers. The ICAS-based stackable switching device 950provides total uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect toexternal network, facilitates non-blocking 1:1 subscription ratio andprovides full mesh non-blocking interconnect with a total of 1.92 Tbpsof switching bandwidth.

ICAS-based stackable switching device has the benefits of improvednetwork congestion, saving the costs, power consumption and spacesavings than the switching devices implemented in the state of the artdata center. As shown in the “ICAS+Stackable Chassis” column of Table 5,data center with ICAS and ICAS-based stackable switching device performsremarkably on data center network with total switching ASIC saving by53.5%, total power consumption saving by 26.0%, total space saving by25.6% and much improved network congestion. However total QSFPtransceiver usage is increased by 2.3%.

The above stackable switching device is for illustrative purpose. Aperson experienced in the art can easily expand the scalability of thestackable switching device and should not be limited as in theillustration.

The stackable switching device addresses the insufficiency in the numberof ports of network switching chip, thus making possible a flexiblenetwork configuration. However, a considerable number of connectingcables and conversion modules have to be used to interconnect theICAS-based interconnection devices and the switching devices. To furtherreduce the use of cables and conversion modules, ICAS modules and switchchips can be electronically interconnected using a PCB and connectors,which is exactly how the multi-unit switching device is structured.Specifically, the ICAS module of the ICAS-based multi-unit switchingdevice is electrically implemented as circuits, and the port groups ofthe ICAS module are soldered or crimped onto a PCB using connectors thatsupport high-speed differential signals and impedance matching. Theinterconnection between the internal port groups is realized using acopper differential pair on the PCB. Since signal losses varysignificantly between different grades of high-speed differentialconnectors and between copper differential pairs on different grades ofPCBs, an active chip can be added at the back end of the connector torestore and enhance the signal to increase the signal transmissiondistance on the PCB. The ICAS module of the ICAS-based multi-unitswitching device may be implemented on a PCB called a fabric card, or ona PCB called a backplane. The copper differential pair on the PCBinterconnects the high-speed differential connector on the PCB to form afull mesh connectivity in the ICAS architecture. The switch chips andrelated circuits are soldered onto a PCB called a line card, which isequipped with a high-speed differential connector docking to the adapteron the fabric card. A multi-U chassis of the ICAS-based multi-unitswitching device includes a plurality of ICAS fabric cards, a pluralityof line cards, and one or two MCU- or CPU-based control cards, one ormore power modules and cooling fan modules. “Rack unit” (“RU” or “U” forshort) measures the height of a data center chassis, equal to 1.75inches. A complete rack is 48 U (48 rack units) in height.

One embodiment of the present invention also provides a chassis-basedmulti-unit (rack unit) switching device. A multi-unit chassis switchingdevice groups multiple switch ICs onto multiple line cards.Chassis-based multi-unit switching equipment interconnects with linecards, control cards, and CPU cards via PCB-based network cards orbackplanes, which saves the cost of transceivers, fiber optic cable andrack space required for interconnection.

Details of an ICAS-based multi-unit chassis switching device 970 areshown in FIG. 9f . FIG. 9f shows 4 ICAS-based fabric cards 97-0 to 97-3interconnected in a full mesh topology to switching ASIC's 98-0 to 98-3.Switching ASIC 98-0 and 98-1 are housed in line card 973, and switchingASIC's 98-2 and 98-3 are housed in line card 974. Line cards 973 and 974are connected through high speed PCB (printed circuit board) connectorsto fabric cards 97-0 to 97-3. In one embodiment, 4 Trident-II ASIC-basedswitches 98-0 to 98-3, each having a switching bandwidth of 24 QSFPs in10 G mode provided in 1:1 subscription ratio, and 4 ICAS-based fabriccards 97-0 to 97-3 containing 3 duplicate copies of ICAS1X5 sub-modulesand each sub-module providing 4×10 G of uplink 971 may be used to buildsa multi-unit chassis switch, as shown in FIG. 9f . Two line cardsprovide data ports 972 of total 1.92 Tbps of bandwidth to connect toservers. ICAS-based multi-unit chassis switching device 970 providestotal uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect to externalnetwork, facilitates full mesh non-blocking 1:1 subscription ratiointerconnect with a total of 1.92 Tbps of switching bandwidth.

Multi-unit chassis-based switching device with fabric cards that areICAS-based full mesh topology has the benefits of improved networkcongestion, saving the costs and power consumption than that ofASIC-based fabric cards implementation with fat tree topology. As shownin the “ICAS+Multi-unit Chassis” column of Table 5, data center withICAS and ICAS-based multi-unit chassis-based switching device performsremarkably on data center network with total QSFP transceiver saving by12.6%, total switching ASIC saving by 53.5%, total power consumptionsaving by 32.7%, total space saving by 29.95% and much improved networkcongestion.

The above multi-unit chassis switching device is for illustrativepurpose. A person experienced in the art can easily expand thescalability of the multi-unit chassis switching device and should not belimited as in the illustration.

The multi-unit chassis-based switching device has the disadvantage of amuch longer development time and a higher cost to manufacture due to itssystem complexity, and is also limited overall by the form factor of themulti-unit chassis. The multi-unit chassis-based switching device,though provides a much larger port count than the single-chip switchingdevice. Although the stackable switching device requires additionaltransceivers and cables than that of the multi-unit chassis-basedapproach, the stackable switching device approach has the advantage ofgreater manageability in the internal network interconnection, virtuallyunlimited scalability, and requires significantly less time forassembling a much larger switching device.

The material required for (i) the data center networks of FIG. 2a ,using state of the art multi-unit switching device (“Fat tree+Multi-unitChassis”), (ii) an implementation of data center network 900 of FIG. 9a, using ICAS-based multi-unit switching device “ICAS+Multi-unitChassis”, and (iii) an implementation of data center network 900 of FIG.9a , using ICAS-based stackable switching device “ICAS+StackableChassis” are summarized and compared in Table 5.

TABLE 5 Fat tree + ICAS + ICAS + Multi-unit Multi-unit Stackable ChassisChassis Chassis Intralink (within Pod) N/A 5 5 Interlink (Across Pod) 45 5 Downlink (to Server) 12 15 15 Total 16 25 25 D:U ratio 3 3 3 D:Iratio N/A 3 3 Number of 10G Interface 96 184.3 184.3 (for comparison)QSFP XCVR Module (Watt) 4 4 4 TOR Switch (Watt) 150 200 200 Multi-unitChassis (Watt) 1660 0 0 Spine-side Interlink QSFP XCVR 18432 18800 38000TOR-side Interlink QSFP XCVR 18432 18800 18800 Fabric / TOR-sideIntralink 36864 18800 18800 QSFP XCVR Server-side QSFP XCVR 55296 5640056400 Total QSFP XCVR 129024 112800 132000 (12.6%) (−2.3%) ASIC in SpineSwitch 2304 1600 1600 ASIC in Fabric Switch 4608 0 0 ASIC in TOR Switch4608 3760 3760 Total Switching ASIC 11520 5360 5360 (53.5%) (53.5%)Spine Switch (KW) 392.448 327.2 472.0 Fabric Switch (KW) 784.896 0 0 TORSwitch (KW) 986.112 1128.0 1128.0 Total Power Consumption (KW) 2163.4561455.2 1600 (32.7%) (26.0%) 96 × QSFP Spine Switch (8U) 1536 0 0 96 ×QSFP Fabric Switch (8U) 3072 0 0 48 × QSFP Spine Switch (4U) 0 1600 1600TOR Switch (1U) 4608 3760 3760 ICAS1X5TRIPLE (1U) 0 0 400 ICAS5X21 (2U)0 376 376 Transpose Rack (36U) 0 720 720 ICAS2X9 (1U) 0 0 0 ICAS8X33(4U) 0 0 0 ICAS10X41(6U) 0 0 0 ICAS16X65 (16U) 0 0 0 Total Rack Unit (U)9216 6456 6856 (29.95%) (25.6%) Pod Interlink Bandwidth (Tbps) 7.7 4.04.0 Pod Intralink Bandwidth (Tbps) 7.7 4.0 4.0 Total Data Link Bandwidth(Pbps) 2.2 2.2 2.2 Per Plane Uplink Bandwidth (Tbps) 7.7 / plane 0 0Total Spine Uplink Bandwidth (Tbps) 0 150.4 601.6 Total ICAS UplinkBandwidth (Tbps) 0 37.6 37.6 Spine-side Interlink QSFP Cable 18432 1880018800 QSFP Fanout Cable (Transpose Rack) 0 37600 37600 QSFP Fanout Cable(ICAS5X21) 0 19740 19740 TOR-side Interlink QSFP Cable 0 18800 18800TOR-side Intralink QSFP Cable 18432 18800 18800 Spine Switch QSFP Cable0 0 19200 Fanout Cable (ICAS1X5TRIPLE) 0 0 19200 Total QSFP Cable 3686456400 75600 Total QSFP Fanout Cable 0 57340 76540

As shown in Table 5, the ICAS-based systems require significantly lesspower dissipation, ASICs and space, resulting in reduced material costsand energy.

The above detailed description is provided to illustrate specificembodiments of the present invention and is not intended to be limiting.Numerous modifications and variations within the scope of the presentinvention are possible. The present invention is set forth in theaccompanying claims.

What is claimed is:
 1. A fanout cable transpose rack supporting kswitches and p interlinks, wherein the fanout cable transpose rackcomprises: k groups of first fiber adapters, each adapter of the kgroups of the first fiber adapters comprising m interfaces, wherein thek groups of the first fiber adapters connect to corresponding ones ofthe k switches through k groups of first fiber cables, wherein the kgroups of the first fiber adapters also connect to an fiber adaptermounting panel by k groups of first fanout fiber cables, wherein each ofa group of ┌p/m┐ first fiber adapters connect to a corresponding groupof ┌p/m┐ fiber adapters of each switch by ┌p/m┐ first fiber cables,wherein each of the group of ┌p/m┐ first fiber adapters connect to thefiber adapter mounting panel by a group of ┌p/m┐ first fanout fibercables, wherein ┌ ┐ is a ceiling function; and p groups of second fiberadapters, each adapter of p groups of the second fiber adapterscomprising m interfaces, wherein the p groups of the second fiberadapters connect p groups of second fiber cables to form p groups ofinterlinks, wherein the p groups of the second fiber adapters alsoconnect to the fiber adapter mounting panel by p groups of second fanoutfiber cables, wherein each of a group of ┌k/m┐ second fiber adaptersconnects to a group of ┌k/m┐ second fiber cables to form an interlink,wherein each of the groups of the ┌k/m┐ second fiber adapters connectsto the fiber adapter mounting panel by a group of ┌k/m┐ second fanoutfiber cables, wherein ┌ ┐ is a ceiling function; wherein the fiberadapter mounting panel, the k groups of the first fanout fiber cablesand the p groups of the second fanout fiber cables are cross-connectedon the fiber adapter mounting panel, through cross-connection, theconnections from the k switches are grouped into p interlinks, eachinterlink containing one connection from each of the k switches, with atotal of k connections per interlink.
 2. A spine plane comprising thefanout cable transpose rack according to claim 1, wherein the spineplane further comprises a plurality of third layer switching devices;wherein the fanout cable transpose rack connects to the plurality ofthird layer switching devices through a plurality of fiber cables; andwherein the fanout cable transpose rack comprises a plurality ofinterlinks in the fanout cable transpose rack to connect to a pluralityof network pods; wherein connections from the plurality of third layerswitching devices are grouped into the plurality of interlinks in thefanout cable transpose rack through the fanout cable transpose rack; andwherein each of the plurality of interlinks in the fanout cabletranspose rack contains one connection from one of the third layerswitching devices, with each interlink in the fanout cable transposerack having a number of connections, wherein the number equals to anumber of the third layer switching devices.